Data extraction in e-commerce

Dissertação de mestrado, Engenharia Eléctrica e Electrónica, Instituto Superior de Engenharia, Universidade do Algarve, 2016Eletronic commerce, know as e-commerce, is a system that consists in buying and selling produtcs/services over the internet. The internet is used by millions of people, making the management of the available information (e.g. competitor analysis market) a very difficult task for those operationg an e-commerce business. So that the managers can better position their companies against competitors, comes the need to create automatic mechanisms to extract information from various web sources (websites). The hotel business is a market where e-commerce is essential since the internet is their biggest selling point, either through sales channels or through their own websites. At the same time, these channels have important information, regarding the reputation of the hotel and their competitors, for instance in the form of guest comments. In this thesis a solution to some of those problems is presented, in which the main focus is the automatic extraction of information from sales channels, such as Booking. com. The extracted information is used to help the hoteliers in the analysis of the prices and opinions of hotel’s guests. That information will be extracted using web robots, able to analyze and interact with web pages, by simulating human behavior. This behavior simulation takes advantage of the navigation patterns present on most sales channels, so that users can easily follow the steps to the final purchase. Briefly describing the overall process, the web robot begins by filling the web site search form with a set of configurable parameters. For each hotel that met the search criteria the most relevant information is extracted, such as: prices, offers, comments and location of the hotel. The collected data is grouped and stored in an intermediate database. Once collected, the data is: (a) used by mathematical prediction models that analyze the prices of the hotels in recent years and generate a forecast of prices that hotels will practice in the future and, (b) used to check the hotel’s reputation taking into account the comments of the guests. This thesis presents a set of four papers resulting in past from the author’s work in project "SRM: Smart Revenue Management" financed by QREN I&DT, no. 38962, with promotor VISUALFORMA - Tecnologias de Informação, SA and co-promoter University of the Algarve.A simplicidade do protocolo HTTP [19] e a extrema flexibilidade dos navegadores web (clientes HTTP) potenciaram o crescimento do número de sites e por sua vez o comércio eletrónico. O comércio eletrónico, também conhecido como e-commerce, é um sistema que consiste na compra e venda de produtos ou serviços através da internet [22]. Sendo a internet um meio de comunicação utilizado por milhões de pessoas, a gestão da informação que é disponibilizada e a análise do mercado concorrente torna-se uma tarefa bastante árdua para quem gere um negócio de e-commerce. Para que os gestores se possam posicionar melhor perante os concorrentes surge a necessidade de criar mecanismos automáticos capazes de extrair informação das várias fontes web (websites). A hotelaria é um mercado em que o e-commerce é imprescindível fazendo da internet o seu maior ponto de venda, seja através de canais de venda ou através dos seus próprios websites. Em simultâneo, os referidos canais apresentam informações importantes sobre a forma de comentários dos hóspedes, relativamente à reputação do hotel e seus concorrentes. Existem dois métodos principais para a procura de informação na web [93], sendo esses: (a) a extração manual através de cópia e colagem e a (b) extração automática através de web robots. Relativamente à extração manual, algumas empresas contratam pessoas para efetuar a extração manual dos dados. Este método consiste em procurar pela web e copiar/colar ficheiros, reformatar texto, imagens, documentos, ficheiros multimédia e outros dados. Este método de extração de dados torna-se dispendioso, pois exige bastante tempo e mão de obra. Por outro lado, para efetuar a extração de dados da web automaticamente, é necessário um crawler (web robot) para visitar as várias páginas web existentes, partindo de uma URL semente. À medida que estas URLs vão sendo visitadas pelo crawler, extraiem-se os dados da página HTML correspondente. Posteriormente por norma esses dados são armazenados numa base de dados, de forma a tornar o acesso aos dados mais eficiente. Nesta dissertação é apresentada uma solução para alguns problemas apresentados, em que o principal foco é a extração automática de informação de quatro canais de venda de reservas de alojamento, sendo esses Booking.com, Tripadvisor, Expedia e Bestday. A informação que se pretende extrair tem como função auxiliar os gestores hoteleiros a analisar a disponibilidade de quartos, os preços praticados e a opinião dos hóspedes relativamente aos hotéis concorrentes. Essa informação será extraída com recurso a web robots, capazes de analisar HTML e interagir com as páginas web simulando o comportamento humano. Esta simulação de comportamento tira partido dos canais de venda seguirem um padrão de navegação de modo a que o utilizador siga facilmente os passos até efetuar a compra. Por cada um dos canais de venda que se pretende extrair informação foi criado um web robot diferente, pois as páginas web estão estruturadas de maneira diferente. Descrevendo sucintamente o processo global, cada web robot começa por efetuar a pesquisa no formulário do respetivo website com um conjunto de parâmetros que são configuráveis. Após efetuar a pesquisa, são percorridos todos os hotéis que satisfizeram os critérios previamente definidos e de seguida é extraída a informação presente nos canais de venda, como sejam: os preços, as ofertas, os comentários e a localização do hotel. Esses dados são agrupados e armazenados numa base de dados não relacional. Nesta fase os dados armazenados estão em bruto, i.e., sem qualquer tratamento. Posteriormente, num processo independente (assíncrono), esses dados serão consolidados através de algumas regras previamente definidas de modo a eliminar redundância e a aumentar a consistência dos mesmos. Neste processo de consolidação existem várias preocupações, sendo possivelmente a principal a associação dos dados extraídos das diferentes páginas. Esta problemática surge devido à discrepância dos nomes dos hotéis nos diferentes canais de vendas. Além disso existem muitas outras discrepâncias entre os canais sendo as mais importantes: o número de estrelas das unidades hoteleiras, o nome dos quartos e a escala de pontuação dos hóspedes. Após concluído todo este processo de tratamento da informação, os dados são armazenados numa base de dados final. Ao contrário da base de dados usada na primeira fase, esta é uma base de dados relacional, o que significa que os dados estão devidamente estruturados possibilitando assim o uso por vários tipos de aplicações. Depois de recolhidos e consolidados, a finalidade dos dados é serem: (a) Utilizados por modelos de previsão matemáticos que analisam os preços praticados pelos hotéis nos últimos anos e geram uma previsão de preços que os hotéis irão praticar no futuro, e (b) utilizados para verificar a reputação dos hotéis tendo em conta os comentários dos hóspedes. Este trabalho não só apresenta a implementação dos web robots e da construção dos dados, como também uma vertente de análise da reputação dos hotéis através da análise dos comentários e pontuação dos hóspedes. A análise desses comentários e pontuações consiste em aplicar algumas regras de semântica e algumas métricas de modo a entender quais são os índices de satisfação dos hóspedes dos hotéis. Através destes indíces é possível verificar a importância de um hotel no mercado, pois num negócio são os clientes que definem o seu sucesso. Esta dissertação apresenta um conjunto de quatro artigos resultantes em parte do trabalho desenvolvido pelo autor no projeto “SRM: Smart Revenue Management” financiado pelo QREN I&DT, n.º 38962, promotor VISUALFORMA - Tecnologias de Informação, SA e co-promotor Universidade do Algarve. Abaixo segue-se a listagem dos artigos que compoem este trabalho: • Martins, D., Lam, R., Rodrigues, J.M.F., Cardoso, P.J.S., Serra, F. (2015) A Web Crawler Framework for Revenue Management, In Proc. 14th Int. Conf. on Artificial Intelligence, Knowledge Engineering and Data Bases (AIKED ’15), in Advances in Electrical and Computer Engineering, Tenerife, Canary Islands, Spain, 10-12 Jan, pp. 88-97. ISBN: 978-1-61804-279-8. • Ramos, C.M.Q., Correia, M.B., Rodrigues, J.M.F., Martins, D., Serra, F. (2015) Big Data Warehouse Framework for Smart Revenue Management. In Proc. 3rd NAUN Int. Conf. on Management, Marketing, Tourism, Retail, Finance and Computer Applications (MATREFC ’15), in Advances in Environmental Science and Energy Planning, Tenerife, Canary Islands, Spain, 10-12 Jan., pp. 13-22. ISBN: 978-1-61804-280-4. • Martins, D., Ramos, C.M.Q, Rodrigues, J.M.F., Cardoso, P.J.S., Lam, R., Serra, F. (2015) Challenges in Building a Big Data Warehouse Applied to the Hotel Business Intelligence, In Proc. 6th Int. Conf. on Applied Informatics and Computing Theory (AICT’15), in Recent Research in Applied Informatics, Salerno, Italy, 27-29 June, pp. 110-117. ISBN: 978-1-61804-313-9. • Choupina, R., Correia, M.B., Ramos, C.M.Q, Martins, D., Serra, F. (2015) Guest Reputation Indexes to Analyze the Hotel’s Online Reputation Using Data Extracted from OTAs, in Proc. 6th Int. Conf. on Applied Informatics and Computing Theory (AICT’15), in Recent Research in Applied Informatics, Salerno, Italy, 27-29 June, pp. 50-59 ISBN: 978-1-61804-313-9

Pervasive gaps in Amazonian ecological research

Biodiversity loss is one of the main challenges of our time,1,2 and attempts to address it require a clear un derstanding of how ecological communities respond to environmental change across time and space.3,4 While the increasing availability of global databases on ecological communities has advanced our knowledge of biodiversity sensitivity to environmental changes,5–7 vast areas of the tropics remain understudied.8–11 In the American tropics, Amazonia stands out as the world’s most diverse rainforest and the primary source of Neotropical biodiversity,12 but it remains among the least known forests in America and is often underrepre sented in biodiversity databases.13–15 To worsen this situation, human-induced modifications16,17 may elim inate pieces of the Amazon’s biodiversity puzzle before we can use them to understand how ecological com munities are responding. To increase generalization and applicability of biodiversity knowledge,18,19 it is thus crucial to reduce biases in ecological research, particularly in regions projected to face the most pronounced environmental changes. We integrate ecological community metadata of 7,694 sampling sites for multiple or ganism groups in a machine learning model framework to map the research probability across the Brazilian Amazonia, while identifying the region’s vulnerability to environmental change. 15%–18% of the most ne glected areas in ecological research are expected to experience severe climate or land use changes by 2050. This means that unless we take immediate action, we will not be able to establish their current status, much less monitor how it is changing and what is being lostinfo:eu-repo/semantics/publishedVersio

ATLANTIC EPIPHYTES: a data set of vascular and non-vascular epiphyte plants and lichens from the Atlantic Forest

Epiphytes are hyper-diverse and one of the frequently undervalued life forms in plant surveys and biodiversity inventories. Epiphytes of the Atlantic Forest, one of the most endangered ecosystems in the world, have high endemism and radiated recently in the Pliocene. We aimed to (1) compile an extensive Atlantic Forest data set on vascular, non-vascular plants (including hemiepiphytes), and lichen epiphyte species occurrence and abundance; (2) describe the epiphyte distribution in the Atlantic Forest, in order to indicate future sampling efforts. Our work presents the first epiphyte data set with information on abundance and occurrence of epiphyte phorophyte species. All data compiled here come from three main sources provided by the authors: published sources (comprising peer-reviewed articles, books, and theses), unpublished data, and herbarium data. We compiled a data set composed of 2,095 species, from 89,270 holo/hemiepiphyte records, in the Atlantic Forest of Brazil, Argentina, Paraguay, and Uruguay, recorded from 1824 to early 2018. Most of the records were from qualitative data (occurrence only, 88%), well distributed throughout the Atlantic Forest. For quantitative records, the most common sampling method was individual trees (71%), followed by plot sampling (19%), and transect sampling (10%). Angiosperms (81%) were the most frequently registered group, and Bromeliaceae and Orchidaceae were the families with the greatest number of records (27,272 and 21,945, respectively). Ferns and Lycophytes presented fewer records than Angiosperms, and Polypodiaceae were the most recorded family, and more concentrated in the Southern and Southeastern regions. Data on non-vascular plants and lichens were scarce, with a few disjunct records concentrated in the Northeastern region of the Atlantic Forest. For all non-vascular plant records, Lejeuneaceae, a family of liverworts, was the most recorded family. We hope that our effort to organize scattered epiphyte data help advance the knowledge of epiphyte ecology, as well as our understanding of macroecological and biogeographical patterns in the Atlantic Forest. No copyright restrictions are associated with the data set. Please cite this Ecology Data Paper if the data are used in publication and teaching events. © 2019 The Authors. Ecology © 2019 The Ecological Society of Americ

Pervasive gaps in Amazonian ecological research

Biodiversity loss is one of the main challenges of our time,1,2 and attempts to address it require a clear understanding of how ecological communities respond to environmental change across time and space.3,4 While the increasing availability of global databases on ecological communities has advanced our knowledge of biodiversity sensitivity to environmental changes,5,6,7 vast areas of the tropics remain understudied.8,9,10,11 In the American tropics, Amazonia stands out as the world's most diverse rainforest and the primary source of Neotropical biodiversity,12 but it remains among the least known forests in America and is often underrepresented in biodiversity databases.13,14,15 To worsen this situation, human-induced modifications16,17 may eliminate pieces of the Amazon's biodiversity puzzle before we can use them to understand how ecological communities are responding. To increase generalization and applicability of biodiversity knowledge,18,19 it is thus crucial to reduce biases in ecological research, particularly in regions projected to face the most pronounced environmental changes. We integrate ecological community metadata of 7,694 sampling sites for multiple organism groups in a machine learning model framework to map the research probability across the Brazilian Amazonia, while identifying the region's vulnerability to environmental change. 15%–18% of the most neglected areas in ecological research are expected to experience severe climate or land use changes by 2050. This means that unless we take immediate action, we will not be able to establish their current status, much less monitor how it is changing and what is being lost

Rationale, study design, and analysis plan of the Alveolar Recruitment for ARDS Trial (ART): Study protocol for a randomized controlled trial

Background: Acute respiratory distress syndrome (ARDS) is associated with high in-hospital mortality. Alveolar recruitment followed by ventilation at optimal titrated PEEP may reduce ventilator-induced lung injury and improve oxygenation in patients with ARDS, but the effects on mortality and other clinical outcomes remain unknown. This article reports the rationale, study design, and analysis plan of the Alveolar Recruitment for ARDS Trial (ART). Methods/Design: ART is a pragmatic, multicenter, randomized (concealed), controlled trial, which aims to determine if maximum stepwise alveolar recruitment associated with PEEP titration is able to increase 28-day survival in patients with ARDS compared to conventional treatment (ARDSNet strategy). We will enroll adult patients with ARDS of less than 72 h duration. The intervention group will receive an alveolar recruitment maneuver, with stepwise increases of PEEP achieving 45 cmH(2)O and peak pressure of 60 cmH2O, followed by ventilation with optimal PEEP titrated according to the static compliance of the respiratory system. In the control group, mechanical ventilation will follow a conventional protocol (ARDSNet). In both groups, we will use controlled volume mode with low tidal volumes (4 to 6 mL/kg of predicted body weight) and targeting plateau pressure <= 30 cmH2O. The primary outcome is 28-day survival, and the secondary outcomes are: length of ICU stay; length of hospital stay; pneumothorax requiring chest tube during first 7 days; barotrauma during first 7 days; mechanical ventilation-free days from days 1 to 28; ICU, in-hospital, and 6-month survival. ART is an event-guided trial planned to last until 520 events (deaths within 28 days) are observed. These events allow detection of a hazard ratio of 0.75, with 90% power and two-tailed type I error of 5%. All analysis will follow the intention-to-treat principle. Discussion: If the ART strategy with maximum recruitment and PEEP titration improves 28-day survival, this will represent a notable advance to the care of ARDS patients. Conversely, if the ART strategy is similar or inferior to the current evidence-based strategy (ARDSNet), this should also change current practice as many institutions routinely employ recruitment maneuvers and set PEEP levels according to some titration method.Hospital do Coracao (HCor) as part of the Program 'Hospitais de Excelencia a Servico do SUS (PROADI-SUS)'Brazilian Ministry of Healt

Reducing the environmental impact of surgery on a global scale: systematic review and co-prioritization with healthcare workers in 132 countries

Abstract Background Healthcare cannot achieve net-zero carbon without addressing operating theatres. The aim of this study was to prioritize feasible interventions to reduce the environmental impact of operating theatres. Methods This study adopted a four-phase Delphi consensus co-prioritization methodology. In phase 1, a systematic review of published interventions and global consultation of perioperative healthcare professionals were used to longlist interventions. In phase 2, iterative thematic analysis consolidated comparable interventions into a shortlist. In phase 3, the shortlist was co-prioritized based on patient and clinician views on acceptability, feasibility, and safety. In phase 4, ranked lists of interventions were presented by their relevance to high-income countries and low–middle-income countries. Results In phase 1, 43 interventions were identified, which had low uptake in practice according to 3042 professionals globally. In phase 2, a shortlist of 15 intervention domains was generated. In phase 3, interventions were deemed acceptable for more than 90 per cent of patients except for reducing general anaesthesia (84 per cent) and re-sterilization of ‘single-use’ consumables (86 per cent). In phase 4, the top three shortlisted interventions for high-income countries were: introducing recycling; reducing use of anaesthetic gases; and appropriate clinical waste processing. In phase 4, the top three shortlisted interventions for low–middle-income countries were: introducing reusable surgical devices; reducing use of consumables; and reducing the use of general anaesthesia. Conclusion This is a step toward environmentally sustainable operating environments with actionable interventions applicable to both high– and low–middle–income countries

Mortality from gastrointestinal congenital anomalies at 264 hospitals in 74 low-income, middle-income, and high-income countries: a multicentre, international, prospective cohort study

Summary Background Congenital anomalies are the fifth leading cause of mortality in children younger than 5 years globally. Many gastrointestinal congenital anomalies are fatal without timely access to neonatal surgical care, but few studies have been done on these conditions in low-income and middle-income countries (LMICs). We compared outcomes of the seven most common gastrointestinal congenital anomalies in low-income, middle-income, and high-income countries globally, and identified factors associated with mortality. Methods We did a multicentre, international prospective cohort study of patients younger than 16 years, presenting to hospital for the first time with oesophageal atresia, congenital diaphragmatic hernia, intestinal atresia, gastroschisis, exomphalos, anorectal malformation, and Hirschsprung’s disease. Recruitment was of consecutive patients for a minimum of 1 month between October, 2018, and April, 2019. We collected data on patient demographics, clinical status, interventions, and outcomes using the REDCap platform. Patients were followed up for 30 days after primary intervention, or 30 days after admission if they did not receive an intervention. The primary outcome was all-cause, in-hospital mortality for all conditions combined and each condition individually, stratified by country income status. We did a complete case analysis. Findings We included 3849 patients with 3975 study conditions (560 with oesophageal atresia, 448 with congenital diaphragmatic hernia, 681 with intestinal atresia, 453 with gastroschisis, 325 with exomphalos, 991 with anorectal malformation, and 517 with Hirschsprung’s disease) from 264 hospitals (89 in high-income countries, 166 in middleincome countries, and nine in low-income countries) in 74 countries. Of the 3849 patients, 2231 (58·0%) were male. Median gestational age at birth was 38 weeks (IQR 36–39) and median bodyweight at presentation was 2·8 kg (2·3–3·3). Mortality among all patients was 37 (39·8%) of 93 in low-income countries, 583 (20·4%) of 2860 in middle-income countries, and 50 (5·6%) of 896 in high-income countries (p<0·0001 between all country income groups). Gastroschisis had the greatest difference in mortality between country income strata (nine [90·0%] of ten in lowincome countries, 97 [31·9%] of 304 in middle-income countries, and two [1·4%] of 139 in high-income countries; p≤0·0001 between all country income groups). Factors significantly associated with higher mortality for all patients combined included country income status (low-income vs high-income countries, risk ratio 2·78 [95% CI 1·88–4·11], p<0·0001; middle-income vs high-income countries, 2·11 [1·59–2·79], p<0·0001), sepsis at presentation (1·20 [1·04–1·40], p=0·016), higher American Society of Anesthesiologists (ASA) score at primary intervention (ASA 4–5 vs ASA 1–2, 1·82 [1·40–2·35], p<0·0001; ASA 3 vs ASA 1–2, 1·58, [1·30–1·92], p<0·0001]), surgical safety checklist not used (1·39 [1·02–1·90], p=0·035), and ventilation or parenteral nutrition unavailable when needed (ventilation 1·96, [1·41–2·71], p=0·0001; parenteral nutrition 1·35, [1·05–1·74], p=0·018). Administration of parenteral nutrition (0·61, [0·47–0·79], p=0·0002) and use of a peripherally inserted central catheter (0·65 [0·50–0·86], p=0·0024) or percutaneous central line (0·69 [0·48–1·00], p=0·049) were associated with lower mortality. Interpretation Unacceptable differences in mortality exist for gastrointestinal congenital anomalies between lowincome, middle-income, and high-income countries. Improving access to quality neonatal surgical care in LMICs will be vital to achieve Sustainable Development Goal 3.2 of ending preventable deaths in neonates and children younger than 5 years by 2030

Unraveling the genetic background of individuals with a clinical familial hypercholesterolemia phenotype

Familial hypercholesterolemia (FH) is a common genetic disorder of lipid metabolism caused by pathogenic/likely pathogenic variants in LDLR, APOB, and PCSK9 genes. Variants in FH-phenocopy genes (LDLRAP1, APOE, LIPA, ABCG5, and ABCG8), polygenic hypercholesterolemia, and hyperlipoprotein (a) [Lp(a)] can also mimic a clinical FH phenotype. We aim to present a new diagnostic tool to unravel the genetic background of clinical FH phenotype. Biochemical and genetic study was performed in 1,005 individuals with clinical diagnosis of FH, referred to the Portuguese FH Study. A next-generation sequencing panel, covering eight genes and eight SNPs to determine LDL-C polygenic risk score and LPA genetic score, was validated, and used in this study. FH was genetically confirmed in 417 index cases: 408 heterozygotes and 9 homozygotes. Cascade screening increased the identification to 1,000 FH individuals, including 11 homozygotes. FH-negative individuals (phenotype positive and genotype negative) have Lp(a) >50 mg/dl (30%), high polygenic risk score (16%), other monogenic lipid metabolism disorders (1%), and heterozygous pathogenic variants in FH-phenocopy genes (2%). Heterozygous variants of uncertain significance were identified in primary genes (12%) and phenocopy genes (7%). Overall, 42% of our cohort was genetically confirmed with FH. In the remaining individuals, other causes for high LDL-C were identified in 68%. Hyper-Lp(a) or polygenic hypercholesterolemia may be the cause of the clinical FH phenotype in almost half of FH-negative individuals. A small part has pathogenic variants in ABCG5/ABCG8 in heterozygosity that can cause hypercholesterolemia and should be further investigated. This extended next-generation sequencing panel identifies individuals with FH and FH-phenocopies, allowing to personalize each person’s treatment according to the affected pathway

Ciência, Crise e Mudança. 3.º Encontro Nacional de História das Ciências e da Tecnologia. ENHCT2012

III Encontro Nacional de História das Ciências e da Tecnologia. O Centro de Estudos de História e Filosofia da Ciência, organiza o 3.º Encontro Nacional de História da Ciência e da Técnica, sob o tema «Ciência, Crise e Mudança» que tem lugar na Universidade de Évora, nos dias 26, 27 e 28 de Setembro de 2012. O Primeiro Encontro Nacional de História da Ciência teve lugar em 21 e 22 Julho de 2009, no seguimento do programa de estímulo ao de¬senvolvimento da História da Ciência em Portugal e de valorização do património cultural e científico do País, lançado pelo Ministério da Ciência, Tecnologia e Ensino Superior (MCTES) em 31 de Janeiro desse ano. A sua organização coube a investigadores do Instituto de História Contemporânea (IHC), da FCSH da UNL, e do Centro Científico e Cultural de Macau (CCCM), em cujas instalações se realizou. De en¬tre as conclusões do Encontro, destacou-se a de realizar periodicamen¬te novos Encontros Nacionais, a serem organizados de forma rotativa por diferentes centros e núcleos de investigadores. Na sequência deste Primeiro Encontro, o Centro Interuniversitário de História das Ciências e da Tecnologia (CIUHCT) organizou, entre 26 e 28 de Julho de 2010, o II Encontro, dedicado ao tema “Comunicação das Ciências e da Tecnologia em Portugal: Agentes, Meios e Audiências”. Cabe agora ao CEHFCi cumprir o que foi decidido no final deste Encontro. Na situação económica e política que hoje vivemos torna-se particularmente urgente aprofundar o estudo e o debate sobre a interação entre a Sociedade, a Ciência e a sua História. Coordenação Científica e Executiva do encontro estiveram a cargo de dois investigadores CEHFCi: Maria de Fátima Nunes, José Pedro Sousa Dia